Wine Review Analysis
STAT 420, Summer 2017, Eloise Rosen (eloiser2), Jondean Haley (jahaley2), Nilesh Malpekar (nmalpe2)
8/4/2017
Introduction
In this project we use a dataset of wine reviews to predict review points from numerical, categorical and textual predictors.
The data is from Kaggle Datasets, and covers 150k wine reviews along with some attributes of the wines. It can be found here. (A (free) Kaggle login is required to access it directly from kaggle.com). The data was originally scraped from WineEnthusiast.
The dataset contains the following columns:
Points: the number of points the wine received in its review, on a scale of 1-100. However, only wines with >= 80 points were published in the dataset. We plan to transform this feature and check out logit, probit, BoxCox and log transforms. This will be our response variable.
Description: a description of the wine’s taste, smell, look, feel, etc. We plan to use LDA topic modeling to convert the text description into a small vector of numerical predictors. We plan to generate a “sentiment” numeric predictor using a sentiment classifer. This is an additive predictor.
Price: the cost for a bottle of the wine. We will transform this using log or inverse or a power of the inverse (TBD by CV), or any combination of these.
Variety: the type of grapes used
Country: the country that the wine is from
This is a particularly interesting problem for several reasons:
- We need to transform both predictors and response, which makes for a more interesting problem. We can dig into research of optimal transforms using cross validation over a grid of models.
- The regression via glm shows heavy tails on the errors in the QQ plot. Because we have 137k data points, this offers us an opportunity to argue that by the CLT the heavy tails do not obviate our use of t and F tests. We can explore the pitfalls of heavy tails and try to use the “heavy” package to do a glm with heavy tails. Again, this is more interesting than a more straight-forward problem.
- Chance to use LDA topic modeling and sentiment analysis
Methods
Feature Engineering
- The country was used to compute the continentTopFive, which was added as a categorical feature. This feature is the continent of the country of origin of the wine, with the separation of the top 5 wine producers in Europe into their own class. These include France, Germany, Italy, Spain and Portugal. The levels are indicated in the plot of points vs log(price), by continentTopFive.
- LDA for 2 topics was performed on the description of the wine. The probability of being in the first topic was added as a feature named “topic1”.
The sentiment of the description of the wine was computed using bing. Sentiment was added as a numeric predictor, named “sentiment”, ranging from -6 to 11.
- The resulting predictors after feature enginering are shown below, with ‘points’ as the response.
Because price ranges from 4 to 2300 USD, and is positive, ranging over 3 orders of magnitude, log(price) is used as a predictor, instead of price.
## [1] "points" "price" "continentTopFive"
## [4] "topic1" "sentiment"Training and Test Set Generation
## [1] TRUEModel Selection
Model #1
- The model started with the base of: \(log(points-79.999) \sim log(price)*continentTopFive*sentiment*topic1 + I(log(price))^2 + I(log(price))^3\)
- The optimal model after backward BIC selection was: \(log(points - 79.999) \sim log(price) + continentTopFive + topic1 + log(price):continentTopFive + continentTopFive:topic1\)
##
## studentized Breusch-Pagan test
##
## data: mod1
## BP = 2824.7, df = 20, p-value < 2.2e-16##
## Shapiro-Wilk normality test
##
## data: sample(resid(mod1), 5000)
## W = 0.91738, p-value < 2.2e-16- Outliers, in which the residual had magnitude over 7 points, were removed. Highly influential points were removed and the model was refit. 1283 observations were removed in total.
- 10-fold cross-validation yielded a mean test RMSE of 2.5611389.
Model 2
- The model started with the base of: \(points \sim log(price)*continentTopFive*sentiment*topic1 + I(log(price))^2 + I(log(price))^3\)
- The optimal model after backward BIC selection was: \(points \sim log(price) + continentTopFive + topic1 + log(price):continentTopFive + log(price):topic1 + continentTopFive:topic1\)
##
## studentized Breusch-Pagan test
##
## data: mod2
## BP = 5604, df = 21, p-value < 2.2e-16##
## Shapiro-Wilk normality test
##
## data: sample(resid(mod2), 5000)
## W = 0.99597, p-value = 1.765e-10- Highly influential points were removed and the model was refit. 3422 observations were removed in total.
- 10-fold cross-validation yielded a mean test RMSE of 2.4331344.
Model 3
- The Box-Cox transform of the response was performed to try to counteract the indicated heteroskedasticity.
- The model started with a base of: \(bc_points \sim log(price)*continentTopFive*sentiment*topic1 + I(log(price))^2 + I(log(price))^3\)
- The optimal model after backward BIC selection was: \(bc_points \sim log(price) + continentTopFive + topic1 + log(price):continentTopFive + log(price):topic1 + continentTopFive:topic1\)
##
## studentized Breusch-Pagan test
##
## data: mod3
## BP = 4521.8, df = 21, p-value < 2.2e-16##
## Shapiro-Wilk normality test
##
## data: sample(resid(mod3), 5000)
## W = 0.99349, p-value = 2.522e-14- Highly influential points were removed and the model was refit. 0 observations were removed in total.
- 10-fold cross-validation yielded a mean test RMSE of 2.4554481.
Model 4
- Weighted least squares was performed, with weight being proportional to the inverse of the observed variance of points. The observed variance of residuals peaks at points = 90 and tapers to the ends of the points range. The weight function therefore assumes its minimum at points = 90 and increases at the ends of the range of points.
- The weight function used: \(weights = sqrt(1 / 1 + abs(points -90))\)
- The model started with a base of: \(points \sim log(price)*continentTopFive*sentiment*topic1 + I(log(price))^2 + I(log(price))^3\)
- The optimal model after backward BIC selection was: \(points \sim log(price) + continentTopFive + topic1 + log(price):continentTopFive + log(price):topic1 + continentTopFive:topic1\)
##
## studentized Breusch-Pagan test
##
## data: mod4
## BP = 4464.5, df = 21, p-value < 2.2e-16##
## Shapiro-Wilk normality test
##
## data: sample(resid(mod4), 5000)
## W = 0.99624, p-value = 5.742e-10- Highly influential points were removed and the model was refit. 3117 observations were removed in total.
- 10-fold cross-validation yielded a mean test RMSE of 2.5705648.
Results
- 4 models were analyzed as candidates for optimal prediction of points.
- For each, a base model was constructed and backward BIC selection was used to trim the model.
- As seen by the plots and the BP and Shapiro-Wilk tests, normality of the errors is suspect, and heteroskedasticity is indicated for all models. In particular, the variance seems highest for points around 90, and tapers to the extremes. Note however, that since Shapiro-Wilk in R can take a maximum of 5000 residuals, sampling was required, and this test is then dependent on the choice of random seed used in the sampling.
- Many transforms (the 4 here, and not shown) were attemped in order to overcome these violations of the assumptions of multiple linear regression by least squares minimization.
- 10 fold cross-validation was performed on all models, with mean CV RMSE given as below.
| model | CV RMSE |
|---|---|
| 1 | 2.561 |
| 2 | 2.433 |
| 3 | 2.455 |
| 4 | 2.571 |
- Model #2 wins by minimal CV RMSE.
- The CV RMSE can be used to construct prediction intervals, in lieu of analytic formulae based on normal and homoskedastic errors, as \(\hat{y} \pm 2*2.433\) for the prediction interval.
preds = predict(mod2, newdata=wine_test)
success_ratio = sum(wine_test$points >= preds - 2*2.433 & wine_test$points <= preds + 2*2.433) / nrow(wine_test)- Using Model 2 with this prediction interval resulted in a success ratio of 0.9434985 on the withheld test set. ie This percentage of the withheld test set points were within the prediction interval of Model #2.
Discussion
Appendix
## # A tibble: 65,304 x 3
## topic term beta
## <int> <chr> <dbl>
## 1 1 100 4.664335e-04
## 2 2 100 7.077942e-04
## 3 1 20222030 1.119677e-06
## 4 2 20222030 6.400127e-07
## 5 1 age 2.741178e-03
## 6 2 age 4.352850e-03
## 7 1 ahead 8.223265e-05
## 8 2 ahead 3.273607e-05
## 9 1 background 3.161070e-04
## 10 2 background 1.449376e-04
## # ... with 65,294 more rows
## # A tibble: 270 x 4
## term topic1 topic2 log_ratio
## <chr> <dbl> <dbl> <dbl>
## 1 accent 0.0011319352 1.632377e-03 0.5281827
## 2 acid 0.0101020896 1.089604e-02 0.1091502
## 3 add 0.0015936485 6.447203e-04 -1.3055881
## 4 age 0.0027411779 4.352850e-03 0.6671645
## 5 alcohol 0.0007706188 2.302083e-03 1.5788504
## 6 almond 0.0015819072 2.658711e-04 -2.5728660
## 7 almost 0.0008838961 2.106729e-03 1.2530560
## 8 along 0.0013260939 1.152072e-03 -0.2029525
## 9 alongsid 0.0010304528 9.993428e-05 -3.3661550
## 10 also 0.0036764899 4.080261e-04 -3.1715957
## # ... with 260 more rows